Bayesian Inference, Generative Models, and Probabilistic Computations in Interactive Neural Networks∗

This tutorial provides an introduction to several key concepts relatated to the computations performed by interactive neural networks. Many of these concepts are relied on in the development of an new version of the interactive activation (IA) model of McClelland & Rumelhart (1981) called the multinomial interactive activation (MIA) model (Mirman, Bolger, Kaitan & McClelland, 2013). The IA model might be viewed as a simplified model of interactive processing within perceptual and memory networks, and indeed models based on the IA model and related interactive activation and competition networks (McClelland, 1981) are widespread in psychological research. The development here is intended to connect the intuitive principles of the original IA model with Bayesian ideas, showing how a variant of the original model provides a system for principled probabilistic inference similar to that envisioned in a precursor to the interactive activation model by Rumelhart (1977) and systematized by Pearl (1982). The ideas are also closely related to several of the ideas underlying deep belief networks (Hinton & Salakhutdinov, 2006). Taken together, these models provide a bridge between neurophysiological ideas and cognitive theories, and between probabilistic models of cognition and process-oriented connectionist or parallel-distributed processing models. Thus, this tutorial may prove useful as an introduction for those interested in understanding more about the relationship between a simple form of Bayesian computation and both real and artificial neural networks. We begin by presenting Bayes’ formula as a tool for inferring the posterior probability that some hypothesis is true, given prior knowledge of